2014.43: Release ahoy!

Heya,

last time I checked it was monday and I was timotimo, so it seemed like a post on this here blog was appropriate!

This week, most developers have been traveling, on vacation or busy or stuff like that. I personally just spent a bit of time on a benchmark that I stumbled upon on twitter. PerlJam took great care of this month’s Rakudo release.

But here’s a great piece of news: The remaining YAPC::EU talks have been uploaded to youtube and the recordings of the APW have also been posted on youtube. Among those there’s a bunch of talks by the Perl 6 folks about Perl 6 stuff :)

As for actual code stuff inside and near the core … there’s not much to report. I kind of feel a bit guilty, too! One thing that I should point out is that usev6 has been doing a crazy amount of bug triage (writing tests for “testneeded” bugs and closing them) in the recent days. Way cool!

Let’s see what we can come up with in the next 7 days :)

2014.41 & .42: APW, GLR progress, Perl 6 IDE, …

Hey everyone!

I’m sorry I’ve kept you waiting instead of reporting directly after the APW! At least you didn’t miss actual code being worked on for the GLR. There was a whole lot of discussion, though. I had the privilege to watch and listen and occasionally speak up. These people surely know their stuff! Patrick Michaud has written a post on his blog that summarizes the decisions made and things found out.

I also announced that discussions would happen about what ought to be in Perl 6.0.0 and what should wait a bit still. That hasn’t been done intensively at the APW, but that’s not too worrysome. The rough consensus is that we’ll have NFG (Normalized Form Graphemes, see S99), NSA (Native and Shaped Arrays) and the GLR (Great List Refactor™).

So, let me talk a bit about the APW. Aside from very pleasant socializing, there were of course a whole bunch of Talks. I had planned to present the thing I’m cheekishly calling RoR, but I ended up not writing a single line of Code for that project so far, so instead I started doing a little bit of Game Development. You can find the result on my github repository “gtk_cairo_shooter”. I was quite (pleasantly) surprised when I noticed the performance wasn’t actually terrible. After all, Rakudo Perl 6 is supposedly crazy slow, right? Turns out it’s fast enough, even if you spend about half the time of each frame rendering vector graphics entirely in software …

My second lightning talk was about a little “live coding” tool – very similar to jnthn’s “code golf assistant” – that lets you play with cairo. The code can be found in the same repository that also has my little game.

Of course there were other great talks there as well. It’s a safe bet to say that jnthn’s talk was among the best, but nine’s talk about Inline::Perl5 certainly astounded a whole bunch of people! Instead of going on to list all the talks, I’ll just recommend you wait for the recordings to become available and watch them by yourself. The speakers are certainly better at giving their talk than I am!

During the Hackathon, there was lots of discussion, hacking everywhere, people trying out Rakudo Perl 6 for the first time ever or for the first time in years. Unfortunately I was quite exhausted, so I didn’t end up talking to many of the other attendees :(

One person I did talk to was Tim Bunce, who complained (constructively!) about the quality and amount of documentation we have, as well as the fact that it’s very easy to stumble upon very outdated content via search engines etc. Better “on-ramps” are certainly high on our “most wanted” list for the near future …

After all this talk about APW, I’m realizing you’re following the weekly because it’s a somewhat succint summary of what’s been worked on a weekly basis. So here’s the usual assortment of Perl 6 improvements in bullet point form:

  • lizmat has improved deprecation messages by supplying a “since version $foo” aspect and at the same time deprecated a whole bunch of things (even though for the 2014.10 release these deprecations have been temporarily un-done again). Among others, she removed the ability of IO::Handle to represent an unopened file in this commit (somehow I’ve got the feeling I’ve talked about this one already…)
  • The already deprecated KeySet and KeyBag classes as well as the ucfirst sub and method have been kicked out of Rakudo as well as the .delete and .exists methods on Arrays, Hashes, Sets, …
  • moritz improved the worst-case speed of escaping strings in json by making the Str.trans method a whole bunch more efficient. On a string that’s just ten thousand newlines, the run time went from 13.5s to 4.9s after the first commit out of three.
  • FROGGS fixed parsing version literals that have a more-than-one-digit long major version number. This bug had been chilling out in our RT since Dec 2012…
  • b2gills fixed a bug with the enum constructor – that I stumbled upon when building the Cairo module – where giving an element in the enum a value by using a pair constructor with a quoted name wouldn’t set the counter for the next entries in the enum.
  • atroxaper added a –git-depth option to all the git clone commands in rakudo’s and nqp’s build systems. Especially the nqp repository is pretty darn huge due to all the old stage0 blobs we have in there…
  • brrt and me added some more operations to the JIT, as I’ve longed for higher “jitted percentage” in my shooter game thingie. Among them are savecapture, usecapture, getobjsc, and neg_n (by brrt) as well as sin, cos, tan, asin, acos, atan, atan2, time_n, nativecallinvoke (not as good as it could become, though!), pow_n and sqrt_n (by me). That day, I learned, that the jit C call emitter has to put floating point arguments into different registers than regular registers, causing my trigonometry ops to blow up quite spectacularly …
  • FROGGS worked with an OpenSuSE developer to improve the packagability of MoarVM.
  • jnthn vastly improved performance and correctness of the does and but operator with regards to roles and access to a metaobject’s methods. He also implemented a method “lookup” for metaobjects that will find a method for you that you can use for introspection (rather than invocation).
  • peschwa made the flipflop operator do a smartmatch against $_ instead of just calling .Bool.
  • After I found out that Bool.roll is about 10x slower than 2.rand.Int.Bool, lizmat went ahead and gave pretty much all picks and rolls a nice little overhaul that improves performance across the board. Nice!
  • lizmat also created a “new-from-absolute-path” constructor for internal use that ought to make path-juggling functions a nice bit faster, as well.
  • the “more” method on Supplies has been renamed to “emit” and the “winner” keyword has been renamed to “earliest”.
  • moritz and mouq did quite a bit of work on doc.perl6.org; Thanks, you rock!
  • tony-o started a project called “pandapack”. It currently does almost nothing, but I suppose we can already get excited about what it *may* do in the future :)
  • ugexe built an example .travis.yml that uses rakudobrew to compile a fresh rakudo and test your module.
  • Jnthn put his OO::Monitors and OO::Actors module up on the ecosystem – these are the modules he gave a talk about at the APW.
  • paul cochrane added PerlMongers::Hannover (inspired by Lynn Root’s pyladies lightning talk at EP2014) as his very first Perl 6 module. Welcome aboard! :)

Two more things worth pointing out that don’t quite fit into the bullet list format:

masak has been writing a whole bunch of posts about macros. These are meant to be kind of a “request for comments”, so feel free to point out things you find wrong/lacking/unsound/…

azawawi has been working steadily on improving Farabi6, a web-based IDE for Perl 6. The newest design is inspired by the way the IPython notebook handles code “blobs”. I kind of like it and I’m very eager to see what Farabi6 is going to look like in one or two month’s time and I’m hoping I can get a few patches in as well. I’m thinking it’d be a pretty lovely thing to have in Rakudo::Star! Here’s a screenshot for you:

A screenshot of the Farabi 6 in-browser IDE.

 

Even though this post is already quite a bit longer than the usual ones, I still feel like I’ve forgotten something :\

Ah well, it won’t be the end of the world!

See you around! :)

2014.40:

Hey, it’s timotimo again.

Last week, we decided to make building MoarVM with its JIT by default (thanks carlin!). It seems to be pretty stable, but now we’re going to get much more people trying it out.

Now it’s time to look through what else has been done recently:

  • lizmat has worked on IO::Handle and IO::Path stuff a lot. It used to be possible to have an IO::Handle that corresponds to a folder or be unopened. That means every single method on the IO::Handle first had to check if it refers to a directory or if it needs to open the file first. That was a major time sink, especially for cases like a while loop that gets individual characters from a file. It’s hard to put into words how much work this has been!
  • On top of that, lizmat also sprinkled lots and lots of optimizations all over the path/file related things.
  • FROGGS tracked down a major piece of start-up time wastage: on windows, rakudo took about 0.75 seconds to construct $?USAGE, even if it went unused. The new code is also faster than the old one used to be in case you actually do want to see the $?USAGE.
  • jnthn implemented the SUPERSEDE and DECLARE cases of EXPORTHOW.
    • SUPERSEDE lets you override the behavior of declarators like “class” or “grammar” in the scope your module gets imported into to use different metaclasses
    • DECLARE lets you create whole new declarators that act very much like the “class” or “grammar” declarators, like perhaps a “controller” declarator that adds a certain role to all things declared with that directive. (That example is what the initial tests in the spectest suite do)
  • Tony-O added a super simple ORM to the ecosystem called “DB::ORM::Quicky”.

Does that list seem not too impressive to you? Don’t worry, things are going to be exciting: This weekend the APW2014 will take place in Austria and it will be followed by two days of hackathon where many Perl 6 developers are going to be on site. Topics that will be covered include The Great List Refactor and there will surely be discussion of what topics and features to move past the “6.0.0” milestone and what we’ll definitely spec/implement before calling that milestone complete.

I am going to be there, too. I will probably post a weekly that includes the things done at the hackathon and the things presented at the APW on tuesday rather than on monday, as monday is the last day of the hackathon.

In somewhat related news, the yapc eu videos from the Vitosha hall (which is where all the Perl 6 related talks were given) have still not been uploaded. Apparently the setup they used to record the talks is suboptimal and requires a lot of extra work or something like that? Sadly, I don’t have insider information for you :(

Oh and by the way: thanks for all the positive feedback in last week’s comments! I didn’t expect so many people whose names I’ve never seen before to speak up in such a kind manner :)

Looking forward to reporting cool stuff next time :)

2014.38 and 39: Star release and stuff

Hey there,

I finally broke through my little writer’s block phase (well, at least partially) to give you this two-week coverage post! Since it’s already a bit into the third consecutive week, I’m going to cover the 16th through the 28th of last month in this post.

In the first week, masak did a rakudo release and FROGGS did a Rakudo Star release with Parrot and MoarVM and the .msi files for Windows are available already as well.

A nice bit of news is that Rakudo-MoarVM now starts up about 15% faster than perl5 with Moose loaded. We need a different benchmark that puts us back in the second place so that we can improve some more ;)

Actually, on that topic, our performance is improving across the board, but let me break all the changes down in list form:

  • Thanks to lizmat, Supply.lines and Supply.words now exist and the lines method/sub for reading from files or splitting strings now take an “eager” option that gives a little preview of how fast things can get once we have the Great List Refactor™. Other than that, you can also pass a “count” parameter to get the number of lines instead of the lines themselves. Str.words got a similar treatment.
  • lizmat also made sure the output from a Proc::Async’s chars/bytes supplies will not give chunks out of order, which could sometimes happen before.
  • Another neat little thing lizmat added was the beginning of a “dd” sub that dumps variables’ contents together with their name to stderr.
  • There was an attempt to optimize the push method for iterables before, but it unfortunately led to a major performance regression in the variants we test in the benchmark suite. Jnthn now worked some more on that method and made it quite a bit faster than it was before.
  • FROGGS has already done the same optimization to rakudo’s MATCH method that nqp already got a while ago, but even though the initial version of that got optimized by 25% with a later commit, the performance hasn’t caught up to the performance it was before yet. We should reach a pretty good performance improvement after we work out the problems.
  • A big thing people have probably been waiting for is lax mode (not having to predeclare variables before using them). Thanks to FROGGS, you can explicitly enable/disable it with “use strict” or “no strict” (where “use strict” is still the default for scripts), but if you run your code on the commandline with the -e flag, you’ll have lax mode on by default.
  • The nativecallglobal op that lets you access globals exposed by a dynamically loaded C library is now also available on the JVM, thanks FROGGS!
  • Lots of spec test suite work has been put in by lizmat, dwarring, usev6, stmuk, TimToady, …

Here’s things in progress that have not really materialized yet:

  • Jnthn and masak were in China for a dayjob gig and weren’t able to get much coding in, but what they did work on is thinking and discussing a lot about how escape analysis should be implemented on MoarVM.
  • Another topic jnthn put some thought into is how to tackle the improvements to native values. Examples for this are being able to ++ a native int or having an “is rw” accessor for a native attribute in your classes or (which I’m really looking forward to) having native arrays in Perl 6 itself – which would also be a prerequisite to proper vectorization (think SSE) of hyper operations and such.

I would have loved to give you a quite thorough comparison of the performance of rakudo-parrot, rakudo-moar, rakudo-moar-jit and perl5 over more than a year, but across all the versions, all benchmarks seem to have missing data at least once, so the results ended up being “100 points every single release for every compiler, but no benchmarks were run”, which I thought wasn’t very insightful.

Anyway, I hope you’ll excuse the much belated and surprisingly short post; My energy levels seem to be below 0.009 recently :(

Have a nice rest-of-week, esteemed readers, and see you next week!

2014.37: Inline::Perl5, C Globals in NativeCall, …

Not terribly much seemed to happen this week, except that the pretty amazing Inline::Perl5 module progressed a whole lot. Here’s the changes in the list format you’ve likely grown accustomed to:

  • On top of the features you can already see in its readme, Inline::Perl5 can also handle exceptions thrown from p5 code into p6 code and exceptions thrown from p6 code into p5 code using the regular eval/try-catch you’re used to
  • FROGGS added support for “cglobal” to NativeCall, so you can access global values that are offered by libraries.
  • FROGGS also made sure MoarVM works with the latest libuv and bumped the revision.
  • FROGGS also added automated travis builds to MoarVM, NQP and Rakudo.
  • grondilu gave Synopsis 99 an overhaul making sure lots of broken links work now.
  • lizmat subjected Proc::Async to lots of testing and wrote some specs, so that people will have some documentation to read up on when they see the changelog entry for it that we neglected to mention in the last release …
  • lizmat also worked on Str.lines, making it a whole lot faster.
  • thanks to lizmat Str.lines and IO::Handle.lines now take an optional named parameter “eager” that makes them 5x and 3.5x faster respectively. But if you loop over these lines, you’ll have to wait for the whole split to finish before your loop’s body runs for the first time.
  • another thing lizmat added is the %ENV named parameter for Proc::Async to override the environment variables.
  • yet another thing lizmat added is a term π to refer to pi.
  • finally, lizmat made Bags and Mixes give Pairs instead of just the keys when evaluated in list context (like when you “for” over them).
  • cognomial implemented the :args parameter for Grammar.parse.
  • PerlJam built a more efficient version of “samecase” (which transfers upper/lower case information from one string to another)
  • jnthn taught the MoarVM profiler’s allocation counter to also count things allocated by extops (our mechanism for a language implementation (like Rakudo Perl 6) to add some extra opcodes to the VM).
  • moritz transferred mathw’s “Form” module to the perl6 organization on github. retupmoca sent in a pull request to make it pass its tests again. Maybe – now that it’s more visible – more people will feel like hacking on it.
  • krunen added a termios binding to the ecosystem. This allows for nice things like enabling raw input mode …
  • moritz fixed segfaults when trying to send data on a closed socket.
  • TimToady worked a bit on reification (which is asking a lazy list to generate values) to improve performance and fix a certain bug you may have hit if you asked a lazy list to generate a huge number of elements.
  • TimToady also implemented the plan method on lists, which allows you to append a second lazy list to a lazy list without forcing all items to be reified (which is what “,” would do)
  • On our mailing list, Alex Becker gave helpful tips for making Rakudo Star more useful and easier to get into for newcomers and got in a pull request for modules.perl6.org to be more user-friendly.

As for me … I didn’t do anything Perl 6 related last week.

But I may or may not be playing around with thoughts relating to a project I’ll give the abbrevation “RoR”. The rest is highly classified! :P

In related news, TimToady is “studying up” for the Great List Refactor, which is expected to make everything list-related much much faster soon™.

Anyway, I’m really looking forward to more performance tweaks and finally the GLR :)

Have a lovely week!

2014.36: Finalization, lazyness, optimizations.

Hi! It’s timotimo again and this week we’re pretty stoked to present some really neat stuff to you!

  • jnthn implemented finalizer support (giving your classes a DESTROY method or submethod that will be invoked when the object gets collected by the GC) on MoarVM and exposed it to rakudo. Now nine will be in a much better position to work on the Inline::Perl5 module!
  •  Speaking of Inline::Perl5, carlin contributed changes to the module that make it installable via Panda, so it doesn’t require any manual work any more.
  • brrt implemented a little function to force a GC run on MoarVM. It’s not meant for performance tuning, though. Its main purpose is to help write test cases that check the behavior of DESTROY methods and similar things.
  • MoarVM now properly reports errors when mallocing memory fails instead of just crashing, thanks to carlin.
  • carlin also fixed up some cases where signedness vs. unsignedness caused branches to always or never be taken in MoarVM’s guts.
  • brrt taught the JIT log to write out where inlines happened in the code.
  • I implemented bindattrs_* (bind a value to an instance’s attribute given a string that’s not known at compile time) and getattrs_* in the JIT.
  • MoarVM’s GC used to run a full collection through nursery and old generation every 25 runs. Now it counts how much data has been promoted to the old generation since the last time a full collection was done and bases the decision on that fact. I started the patch, jnthn gave it the finishing touches.
  • cognominal worked on the Synopsis 99 a whole lot. If you’re ever puzzled by lingo from the Perl 6 community (for example on this blog), this document lends you a helping hand.
  • The usual assortment of spec test improvements have also happened, of course.
  • In the ecosystem, leont contributed a TAP::Harness module and grondilu added Clifford, a geometric algebra module and btyler built a NativeCall binding to “discount”, a markdown compiler.

Another topic that got a lot of attention by the devs this week is performance of specific things that were currently implemented quite sub-optimally and gave users a very bad experience in common usage scenarios:

  • the .words method of Str used to use the .comb method to do its job. While this makes for very pretty and simple code, it’s far from optimal, especially since we know that whitespace and non-whitespace have to follow each other immediately, TimToady was able to produce an improved version that runs about 6x faster.
  • With well-placed native integers, lizmat was able to improve the speed of List.ACCEPTS (the method that a smartmatch turns into) by about 14% when the check needs to go through the entirety of a big list.
  • jnthn and lizmat went through a few global variables that have costly set-up routines that impact every single start-up. Now they will only be populated with data when they are actually needed.
  • Mouq gave us better code-generation for the -p and -n flags of rakudo.
  • Turning the methods .list, .flat, .eager and .hash of Any into multiple dispatch methods (implemented by lizmat) gives the optimizer better optimization opportunities.
  • lizmat also improved the implementation of “make” (which is used inside grammars and their action classes to attach objects to the match results) to not access the caller’s lexpad more than once. (I actually stumbled upon this opportunity)
  • The Whatever and HyperWhatever objects will no longer have new instances of them created whenever you write * or ** in your code. Instead, a pre-built singleton will be returned by each. This cuts a whole lot of allocations from all over. Especially when iterating over lazy lists you’ll end up calling .gimme(*) and .reify(*) often, where * is just used as a “signal”. These allocations are very small, though. TimToady and jnthn worked on this.
  • jnthn turned take, take-rw, succeed and proceed into multiple dispatch subs, which should impact situations where gather/take is in the hot part of the program and also positively impacts given/when statements.
  • A few micro-optimizations with a small — but definitely measurable — pay-off were done to Str.chomp by lizmat.
  • lizmat began improving IO.lines (makes reading /usr/share/dict/words line-by-line about 32% faster) and moritz added a 4% improvement on top.
  • moritz chopped off two container allocations for every call to ListIter’s reify. On top of that, a few containers in MapIter.reify had to go, as well.
  • By using codepoints instead of one-character-strings inside chomp, jnthn manages to remove some more string allocations in there.
  • jnthn also made throwing exceptions and returning more friendly to the optimizer (in particular, the inliner).
  • In a big bunch of commits, lizmat makes many dynamic variables lazily constructed, removing a significant chunk of the start-up time of every single perl6 program.

All it takes for the devs to start a little performance hack-a-thon is a good, reproducable benchmark with a comparison to another language.

Until The Big List Refactor has happened, the performance of lists, list iteration, lazy lists and so on are going to be suboptimal, as we almost always pay the price for a fully lazy list generation, even when the iteration can be identified to be eager.

Another thing to note is that the name “DESTROY” is not yet final. There’s some discussion about it still, because calling it DESTROY may give people familiar with perl5 a bad idea. As opposed to a reference counted implementation (like CPython or perl5), fully garbage-collected implementations (like PyPy and Rakudo) cannot guarantee that your DESTROY methods are called soon after the object becomes unreachable. The DESTROY method may not even be called at all if the interpreter shuts down before that particular object gets collected. If you want your destruction timely and guaranteed, you’ll have to “do it yourself”.

Either way, it’s been a great week for both performance and feature advances. I’ll be back in a week, or maybe I’ll run benchmarks tonight and just publish an out-of-schedule post here with my results :)

See you around!

2014.35: A Star Release, perl5 interop, …

Hey, timotimo here with another little Perl 6 update!

This time, I can announce an actual Rakudo Star release. FROGGS did the honors this time. The release announcement contains a few ChangeLog entries, but since the previous Star release was in 2014.04 there’s a whole lot of changes you’ll get to enjoy. If compiling the Rakudo Star stuff by yourself seems like too much of a hassle, hoelzro’s got you covered with his Docker image of Rakudo Star.

A few of the YAPC::EU videos have been uploaded to youtube. It seems like these are only the videos from one of the three rooms. None of the Perl 6 related talks happened in that room, though.

After the YAPC::EU, Ovid published a nice blogpost entitled “Try rakudobrew and play with concurrency”.

Other than that, a few things have happened since last week:

  • nine continued work on the “embedding perl5 inside Perl 6″ project. On tuesday he announced he could now use DBI inside perl5 inside rakudo. An important feature for making this work absolutely cleanly is still missing: DESTROY submethods on the Perl 6 side don’t get called by the GC yet, so you’d have to manually manage the Perl5 objects in your program.
  • Jnthn implemented the mechanism to give objects a persistent WHICH when they are used as object keys. This feature is only available on MoarVM, though. On Parrot it is not necessary, as parrot’s GC doesn’t move objects. It will have to be implemented on the JVM backend at some point, though.
  • Jnthn utilized the profiler he built recently to optimize the MATCH method in NQP, which gets called whenever a regex (for example, any token/regex/rule inside a Grammar) matches something and a Match object has to be created. They used to allocate a whole lot of Boxes that were immediately thrown away.
  • jnthn and TimToady worked on making errors like using say without an argument happen at compile-time; it works for much more than just “say”, too!
  • In order to further make matching faster, brrt and I implemented some more ops in the JIT that were blocking frames from being jitted (these include bindattrs_*, indexat and indexnat)
  • lizmat invested a whole lot of time into many different WHICH methods (which are used to create keys for hashes and compare things for equivalence)
  • lizmat also improved the performance of path-related operations and rakudo startup.
  • TimToady implemented many cases of anonymous variables
  • TimToady also started on an implementation of the plan method for lazy lists, which allows you to add a list to the end of a lazy list without forcing it to be eagerly evaluated.
  • hoelzro put more work into the Pod documentation implementation of rakudo.
  • nwc10 improved MoarVM’s compatibility with PowerPC (mostly endian-ness related changes that’d also apply to other architectures)
  • carlin fixed a whole bunch of warnings clang would point out in MoarVM and added arc4random support on platforms that support it (like BSDs)
  • carbin and vendethiel added Crypt::Reop and Sprockets respectively to the ecosystem.
  • masak advocated (and finally implemented) a change to allow *..* as ranges (which mean the same thing as -Inf..Inf)
  • there’s been work on the spec test suite by hoelzro, lizmat, TimToady, jnthn, masak, FROGGS, and Mouq

All in all it’s been a good week. A friend of mine lent me a book about x86 assembly and I’d love to learn what SSE has to offer and maybe implement a simple hacky prototype for arrays of int or num that can be used with SSE instructions or something …

Anyway. Hope you’ll have a nice week :)

2014.34: chugging along, GSoC is over, YAPC::EU, …

This week, there was a Rakudo release (not Rakudo *, which is the distribution that contains the compiler and a bunch of modules) and for someone who usually runs the latest master branch code, seeing the change log is always pretty neat.

Over the weekend the YAPC::EU 2014 happened in Sofia, Bulgaria. There were a few Perl 6 talks as well as lightning talks. I heard the video recordings will be up in “a matter of weeks” … Jnthn did a talk on concurrency and a talk on Rakudo performance advances, in which he revealed he’s been working on a profiler in secret. That was a pretty “wow” moment for pretty much all of the Perl 6 community! There was also a talk by masak about goto and why it’s awesome (no, really) and a cute little lightning talk. On top of that, there were talks by lizmat and lichtkind and a lightning talk by nine, who showed embedding a perl5 interpreter inside Rakudo and embedding Python code inside that. Nifty!

Sadly, I missed many of the talks, so I’ll be patiently (or impatiently) waiting for the recordings to be published.

Anyway, here’s your summarized list of changes during that week:

  • TimToady continued work on making our lazy lists less costly in eager context, like making the xx operator many times faster, so it’s now useful for filling very big arrays or lists with some initial values.
  • Some more performance tuning happened to the X operator (especially for ranges, which you’d idiomatically use to replace a nested for loop to go over 2D coordinates, for example), and the combinations and permutations methods. More is very likely to come soon!
  • Together with brrt, I implemented spesh’d operations as well as jit output for the boolification of iterators; If we already know we’ve created an iterator from an Array or from a Hash, we don’t have to check if the iterator is an array or hash iterator every single time we boolify. And since loops over arrays and hashes in NQP usually end up as a while loop that checks the iterator it creates for its truth value, this ought to show up in a bunch of hot loops.
  • A huge change (which I mentioned above already) is the profiler that got added to MoarVM. You can just run your perl6 or nqp code with –profile or –profile-compilation and get a nice little html output that you can comfortably browse through. Neat!
  • hoelzro and sergot have been working some more on making HTTP::UserAgent style things better/more pleasant/more powerful/more extensible/…

This list is already it, but especially the profiler item is a huge thing. Many of our devs were attending the YAPC::EU event, so they have been watching talks and socializing instead of coding all day and night.

Another thing is, that it’s always quite a bit of work to prepare the release, make sure the test suite remains clean, have your release candidate tarballs tested by enough people, … – I’m glad to say that Coke did a wonderful job this month. I’m hopeful we’ll see a new Rakudo Star release this month, as problems between Parrot and Rakudo that prevented last month’s release have been worked on in the mean time.

Of course, with rakudobrew being readily available as a simple way to install panda (our package manager) for you, the need for a Rakudo Star release is not really that big if you’re willing to put a few minutes rather than a few seconds of work into getting a running Perl 6 compiler.

Another thing: Google Summer of Code is now over and the evaluations have been sent in. Both brrt and sergot have passed and we are extremely pleased with how both projects have turned out so far. In addition, we’ll see more contributions from brrt and sergot in the future :)

Finally, if you’re as impatient as I am, you can have a look at different slide sets that were presented at YAPC::EU.

Anyway, that’s it for today. Hope you’ll have a nice week :)

Weeks 32 and 33:

Last week I got distracted by an ear infection and productivity plummeted, and when I recovered near the end of the week I didn’t want to write the blog post for monday and ignore everything that happened since, nor did I want to write a blog post on friday and have almost nothing to write about the next monday!

So now I’ll give you a blog post covering two weeks instead of just one. Here we go!

  • Mouq has started implementing the ** (called the HyperWhatever), which is like the Whatever Star, except for lists of values rather than individual values (the regular Whatever Star works like this: @names.map: *.uc)
  • Mouq, with input from TimToady, has also started working on making Lists of Lists work to spec, which is an especially nice thing to have in array slices. It means you can have a two-dimensional array and get the second entry of each inner array like @twodim[*;1].
  • Last monday was the “soft pencils down” date for the GSoC. At that day, the moar-jit branch got merged into master, as it’s rather stable now and Configure.pl will build a jit-less MoarVM by default still (use –enable-jit to get a jit-enabled build instead).
  • A whole bunch more ops have been implemented for jitting. Mostly by brrt and Jnthn, but I added a few myself, too. I don’t recall when exactly it happened, but On-Stack-Replacement interacts properly with the JIT now and the JIT can handle extops, too. Handlers, which used to trip up spesh, are functional with spesh and JIT, as well, but I think that has already been the case ~2 weeks ago.
  • The most obvious thing still missing from the jit is support for a bunch of different “param” accessor ops. brrt is still looking for the most efficient way to jit them. After those are done, I expect a tremendous amount of frames that are currently bailing out will be jitted (either “further” or “to completion”).
  • Jnthn implemented asynchronous process spawning and communicating to stdin/stdout/stderr on MoarVM, which is a feature he’ll be giving a talk on during the YAPC::EU in Sofia at the end of this week.
  • Froggs has continued pushing Slang support in Perl 6 further and further. His port of v5 from being NQP code to being pure Perl 6 code is progressing nicely and has just reached the point where the Perl 6 version passes a third of the tests the NQP version used to pass.
  • lizmat nuked eval_dies_ok and eval_lives_ok from the specs, as they are likely to be used incorrectly (local subs and variables are not always available in the eval’d code, as the optimizer is free to turn lexicals into locals and thus hide them from eval). She’s now busy replacing all the uses with direct uses of EVAL in the spectests. A few spectests have already turned out to have been relying on some eval’d code dying, but the code was dying for the wrong reason, thus giving a false positive test result.
  • Jnthn changed both the annotation API and the children node API of the QAST objects, causing fewer allocations (no childrens array for leaf nodes, no annotation hash for annotation-less nodes).
  • Also, Jnthn threw out a “middle man” datastructure that was created and shortly thereafter thrown away and turned into a different datastructure on every single successful match or sub-match. An equivalent change is still needed in rakudo, but having the change in NQP already makes build times better.
  • On top of that, more kinds of things are now lazily deserialized in MoarVM, making start-up faster and cheaper on RAM yet again.
  • sergot posted a nice nearly-end-of-GSoC post on his HTTP::UserAgent and friends project, which includes a bunch of documentation (to be found in the actual HTTP::UserAgent repository itself).
  • I re-ordered code in the library loader that used to cause a whole bunch of stat calls on locations in the libpath for libraries that were already cached! Now the cache is looked at before the filesystem is searched through.
  • japhb has been improving perl6-bench for better windows compatibility and made the history comparison less prone to exaggerated scores from some of the microbenchmarks – the JVM implementation makes really, really short work of empty loops!
  • hoelzro is continuously working on improving the ability to attach documentation to objects like classes, roles, attributes, … in the S26-WHY branch.
  • pmurias is working on a refactor/rewrite of the nqp-js code emitter which leads to better code output, support for “source maps” and other nice things.
  • nwc10 has been a tremendous help for regularly running the newest MoarVM and JIT changes through ASAN (address sanitization) and valgrind for us.
  • lichtkind has resurfaced and worked on the tablets again.
  • TimToady gave pushing a list into another a big speed boost for bigger lists by un-lazying the list that is to be pushed.
  • Another change TimToady did was to give dynamic variables a little cache so that lookups would happen quite a bit quicker and be cheaper in general.
  • donaldh found out what made one of the later stages in Rakudo-JVM’s (and nqp-jvm’s) compilation processes so terribly slow and fixed it. In rakudo’s core setting compilation, stage jast used to take as long as stage parse, now it’s down to about a tenth of that. This impacts every single run of rakudo and nqp, not just the initial build.
  • psch improved some error messages in regexes. Also, some improvements to m:g// and s:g/// are in the pipeline for a later date.
  • PerlJam fixed some problems with floating point numbers and scientific notation in sprintf.

So that’s pretty much it. A whole bunch of nice changes, I must say.

A release is coming up this week, on Thursday if I remember correctly. Unfortunately last month’s Rakudo Star release isn’t finished yet; we’ll see if it will be released later or if the work that has gone into preparing last month’s Star release will just be ported over to the upcoming version directly.

2014.31 (for real this time):

Even though last week’s headline claimed it was about weeks 30 and 31, the 31st week was actually this last one! D’oh! Calendars are hard :)

Anyway, here’s your mostly-weekly fix of changes:

  • Jnthn found a bunch of optimization opportunities in the optimizer (hehe), making it run a bunch faster.
  • Another big win was jnthn finding stuff we did repeatedly or stuff we did just to throw away the results again:
    • When trying to match an <?after …>, the regex compiler would flip the target string every time the <?after > was hit. Now the flipped target string gets cached.
    • Every time an NFA got evaluated (which happens whenever we reach a “|” alternation in a regex or a proto token that has multiple implementations. i.e. very often) we took a closure. Jnthn re-wrote parts of the code that works with the NFA cache and managed to shave off a dozen GC runs from our CORE.setting build!
    • Another 800000 allocations went away after jnthn let the alternation index array be generated statically rather than every time the alternation is hit.
    • Improved handling of sink (which is what handles things like failure objects being thrown and side-effects invoked in certain conditions; only thing you need to know is it gets emitted a whole lot during AST generation) leads to smaller ASTs and gives our code-gen opportunities to output better code.
    • even more improvements I didn’t mention!
  • A little piece of work jnthn suggested I’d do is letting our bytecode specializer only mark a guard (like “please make sure this object is concrete” or “please make sure the type of this object is $FOO”, which would cause a deoptimization if they fail) as necessary, when the optimization that would rely on the guard in question actually was done. Unfortunately, we don’t have before/after measurements for how often specialized bytecode deoptimized …
  • On MoarVM, simple exception throws can now be turned into simple goto operations. This also includes things like redo/continue/last in loops.
  • I implemented specializations for the smart numify and smart stringify operations on MoarVM; Something that happens especially often is numifying lists or hashes, which now turns into a simple elems op (which our specializer can and will further simplify)
  • I also implemented a few very simple ops for the jit, so that brrt could spend more time doing the hard bits.
  • Another thing jnthn worked on is making allocations and deserializations lazy. This helps improve the start-up time of the repl and all programs and also improves the  memory-usage of perl6 programs that don’t use much of the CORE setting, which would probably be most.
  • Vendethiel (Nami-Doc on github) wrote a nice guide to perl6 that became included on learnxinyminutes. Kudos!
  • Froggs and lizmat worked further on CompUnitRepo and module-loading-related things
  • a well-timed “well volunteered!” motivated btyler to start a C binding for the jansson JSON parsing/manipulating/outputting library. In very performance-critical situations, especially when you only use parts of a very big JSON document, this should give you better performance than JSON::Tiny. However, JSON::Tiny is a part of the benchmark suite we’re running all the time, meaning we’ll react to performance regressions switfly and try to figure out what makes it slower than it has to be.

Now here’s some numbers to compare today’s state with the state at the time of the last release:

  • 30 seconds for a NQP build (used to be 37)
  • 57 seconds for a Rakudo build (used to be 1:15)
  • 0.02s and 13 MB maxrss to fire up a Rakudo REPL (used to be 0.04s and 35 MB)
  • 0.2s and 114 MB maxrss for a “say 1″ (used to be 0.27s and 135 MB)
  • now it is: 584.95user 75.88system 3:44.71elapsed 294%CPU for a full spectest run with 4 test_jobs
  • used to be: 765.43user 89.06system 4:40.32elapsed

There’s still more opportunities to deserialize things lazily upon first use in MoarVM which ought to give us lower baseline memory usage and spread out startup time “more evenly” (and remove it when possible).

The baseline memory usage of our Perl 6 compilers has always been something that annoyed me. MoarVM gave us about a 2x memory saving compared to Parrot and now we’ve started work on making the memory usag better. I’m excited! :)

Sadly, brrt has been plagued with very stealthy bugs all week, so progress was kind of slow. What he did manage to finish is support for indexat and jumplist, which is needed in most regexes Also, sp_findmethod, findmeth, findmeth_s, indexat, getdynlex, … are done now, but temporarily commented out for bug-hunting purposes.

That’s it from me for this week. I’m already looking forward for the cool stuff I’ll be able to highlight next week :)