2014.45: Perl 6 gets a release year

So last week Larry Wall made quite a splash when people noticed that the abstract to his talk at FOSDEM 2015 reads “[…] 2015 will be the year that Perl 6 officially launches for production use. […]“. I was taken by surprise as well – which caused this lovely excerpt from the irclog to be passed around on the ‘net – but now that I’ve had a bit of time to think about things, it really does make sense.

At APW2014 during the hackathon it was already rather clear that the main things missing for a “6.0.0″ would be GLR, NSA and NFG; The Great List Refactor™, Native Shaped Arrays and Normalized Form Grapheme™. It would surprise me greatly if it took more than a year with the current amount of volunteer work we’ve got going on.

Another thing is that we’re looking for donations for a new community server to host infrastructure or backups thereof (irc logs, specs, panda metadata, …) and services (evalbot, ecosystem smoking, …). Please see moritz’ article for details if you’ve got money or hardware available that we could make use of!

Anyway, here’s The Week In Bullet Points™:

  • japhb merged the “stress” branch for perl6-bench. It lets us write tests that we expect to put some part of rakudo under so much stress that it may expose crash-bugs or similar stuff.
  • rurban fixed a GC bug in parrot that was preventing panda and some modules from working, thus making a Rakudo Star Parrot release possible again. Also, since the last parrot release, a bunch of tests stopped failing in the spec test suite. They had been fudged in the mean time, so in this status report they show up as “TODO passed”.
  • FROGGS built a Perl 6 slang “Tuxic” (as it was requested by [Tux]) that introduces optional whitespace between the name and parenthesis of a subroutine call.
  • raydiak made his Math::Symbolic module available on the ecosystem. AIUI the “Math::Symbolic” module of perl5 is extremely sophisticated; our version is not there yet, however.
  • ugexe built a module to output lists-of-lists as pretty ASCII tables. It’s called Text::Table::Simple.
  • I wrote a tool that turns text copy-pasted from MoarVM’s “spesh log” into a graphviz-rendered graph that shows control flow (grey arrows between instructions, dotted lines between Basic Blocks), data flow (black arrows between instructions), and related info (deoptimization, Frame Handler, and inlining annotations, Callsite info, …). IMO this makes it much easier to see optimization opportunities. In this particular graph I see — apart from the very obvious overuse of set — a register that gets written to twice in a row (r8(3)), a register that gets written to but apparently never read (r7(5)), and some spesh-to-graphviz bugs :)
  • Farabi6 (azawawi’s browser-based IDE) will now syntax-check your code as you type it and display syntax errors and other problems it finds right inside the code editor widget. Additionally, there’s an initial bit of support for debugging. You can step in, step out, and resume the script you’re writing and the code editor will highlight whatever piece of code is currently active and display partial output in the output tab, as well.
  • “cpandatesters” is now available at the domain http://testers.perl6.org/ – if you want to submit your own test reports, you can check out the branch “reporter” in panda and set the “PANDA_SUBMIT_TESTREPORTS” environment variable to some value. You can also smoke all ecosystem modules by running “panda –exclude=panda smoke”. Sadly, the openpipe instruction on Parrot doesn’t allow setting a working path and environment yet, so sending reports on Parrot is currently not supported.
  • lizmat has been working some more on paths, dir(), chdir semantics etc etc.

So what’s next?

  • FROGGS is improving openpipe on parrot, so that panda reports can work there (and so that the branch can be merged back into master)
  • On top of that, the “by author” list of cpandatesters is NYI :)
  • I’m looking to build a less bare API around SDL2 and make Cairo easier to integrate (if you can even call this “not easy” at all). Also, the SDL2::Raw binding doesn’t cover anywhere near enough ground for a full game.
  • And of course I’m always interested in improving performance across the board!
  • Of course there’s more cool stuff that can be put into farabi :)
  • masak and vendethiel have been passing interesting papers/articles/factoids back and forth about macro systems, so I’m excited to see what our system will end up looking like!

Also I’m hoping the recent announcement about Perl 6′s release in 2015 will bring in some more interested individuals for testing, module porting/development, feature requests, or just cheering us on ;)

See you next week!
- timotimo

2014.44: A continuous trickle of stuff.

Greetings, readers!

Last week saw the creation of the website http://perl6.guide/ (generic top-level-domains for the win!), which seems to have made quite a splash! Other than that, FROGGS has started building a “cpandatesters” web app. It is designed to smoke-test the ecosystem and receive reports from users who opt-in to the “panda report” feature. You can find that feature on the panda branch “report”, but without the service to accept the reports, it’s not that interesting. The reports would look something like this.

Still being frustrated from how hard it is to get an OpenGL-backed cairo renderered surface on GTK3, I started building a raw binding for SDL2. It contains a SDL2 version of the “shoot ‘em up” I made for the APW, but since I procrastinated too much, it doesn’t yet render the player’s ship, explosions, or pretty laser blasts. Oh well. Before anyone attempts to write a game with that binding, be warned: it is very low-level and annoying to use. I’m going to write a more object-oriented binding on top of the existing module (or maybe as a replacement?) in the near future, though.

So, what else has happened? Here’s the list you’ve certainly grown accustomed to:

  • FROGGS implemented the close-pipe method that will give you the exit code of a program you’ve spawned as a subprocess with the “open pipe” method.
  • FROGGS added two-argument candidates for the min and max operators, resulting in a great speedup.
  • FROGGS also made perl6-m and nqp-m write out the profiler’s results when the script was exited with the “exit” sub.
  • I added the ability to nativecast a CStruct to a different CStruct, because I needed that to support SDL2′s Event structures.
  • lizmat spec’d and tested support for a “stopper” in the Scheduler’s cue method. This is a callable that will get called for each repetition of a piece of work that’s been cued to repeat and allows supplying a condition that’d kick that piece of work out. The code itself had been done by jnthn at some point in the past.
  • lizmat has been working on IO::Path. It’s going to turn into IO::Dir and IO::File (which both do the IO::Local) role. Another object may exist for symlinks. A third role will supply all operations for reading and writing. Most operations you may be interested in for file stuff will operate on just strings as paths, as well.
  • TimToady made using $a_string inside regexes much faster and fixed $an_array to stringify when used inside a regex.
  • a somewhat new community member named “jepaway” has been putting a lot of thought into Perl 6′s Date and DateTime stuff as well as timezones. As an early result, DateTime is now friendlier to subclassing.
  • Mouq made rakudo accept method calls directly following a class’ declaration, like in
    class :: { method foo { say "bar" }}.foo
  • The rather peculiar “winner” statement is now called “earliest”, which makes it a bit easier to understand.
  • When playing around with Farabi6, I found a rather painfully slow piece of code responsible for splitting a HTTP request’s header section off from its body. I was able to speed it up by about 300x quite easily. This makes startup times and reaction times of Farabi6 (and all other HTTP::Easy based HTTP Apps) drastically better.
  • On the topic of Farabi6, azawawi started integrating the Rakudo Debugger. So far all it does is give you buttons that proclaim “NYI!”, but I’m quite looking forward to seeing what it’ll look like :)

It hasn’t been a terribly busy week for me, but let’s see if I can’t turn my tiny game into something nice for the next post :)

That’s it for this post. Have a lovely week!

2014.43: Release ahoy!

Heya,

last time I checked it was monday and I was timotimo, so it seemed like a post on this here blog was appropriate!

This week, most developers have been traveling, on vacation or busy or stuff like that. I personally just spent a bit of time on a benchmark that I stumbled upon on twitter. PerlJam took great care of this month’s Rakudo release.

But here’s a great piece of news: The remaining YAPC::EU talks have been uploaded to youtube and the recordings of the APW have also been posted on youtube. Among those there’s a bunch of talks by the Perl 6 folks about Perl 6 stuff :)

As for actual code stuff inside and near the core … there’s not much to report. I kind of feel a bit guilty, too! One thing that I should point out is that usev6 has been doing a crazy amount of bug triage (writing tests for “testneeded” bugs and closing them) in the recent days. Way cool!

Let’s see what we can come up with in the next 7 days :)

2014.41 & .42: APW, GLR progress, Perl 6 IDE, …

Hey everyone!

I’m sorry I’ve kept you waiting instead of reporting directly after the APW! At least you didn’t miss actual code being worked on for the GLR. There was a whole lot of discussion, though. I had the privilege to watch and listen and occasionally speak up. These people surely know their stuff! Patrick Michaud has written a post on his blog that summarizes the decisions made and things found out.

I also announced that discussions would happen about what ought to be in Perl 6.0.0 and what should wait a bit still. That hasn’t been done intensively at the APW, but that’s not too worrysome. The rough consensus is that we’ll have NFG (Normalized Form Graphemes, see S99), NSA (Native and Shaped Arrays) and the GLR (Great List Refactor™).

So, let me talk a bit about the APW. Aside from very pleasant socializing, there were of course a whole bunch of Talks. I had planned to present the thing I’m cheekishly calling RoR, but I ended up not writing a single line of Code for that project so far, so instead I started doing a little bit of Game Development. You can find the result on my github repository “gtk_cairo_shooter”. I was quite (pleasantly) surprised when I noticed the performance wasn’t actually terrible. After all, Rakudo Perl 6 is supposedly crazy slow, right? Turns out it’s fast enough, even if you spend about half the time of each frame rendering vector graphics entirely in software …

My second lightning talk was about a little “live coding” tool – very similar to jnthn’s “code golf assistant” – that lets you play with cairo. The code can be found in the same repository that also has my little game.

Of course there were other great talks there as well. It’s a safe bet to say that jnthn’s talk was among the best, but nine’s talk about Inline::Perl5 certainly astounded a whole bunch of people! Instead of going on to list all the talks, I’ll just recommend you wait for the recordings to become available and watch them by yourself. The speakers are certainly better at giving their talk than I am!

During the Hackathon, there was lots of discussion, hacking everywhere, people trying out Rakudo Perl 6 for the first time ever or for the first time in years. Unfortunately I was quite exhausted, so I didn’t end up talking to many of the other attendees :(

One person I did talk to was Tim Bunce, who complained (constructively!) about the quality and amount of documentation we have, as well as the fact that it’s very easy to stumble upon very outdated content via search engines etc. Better “on-ramps” are certainly high on our “most wanted” list for the near future …

After all this talk about APW, I’m realizing you’re following the weekly because it’s a somewhat succint summary of what’s been worked on a weekly basis. So here’s the usual assortment of Perl 6 improvements in bullet point form:

  • lizmat has improved deprecation messages by supplying a “since version $foo” aspect and at the same time deprecated a whole bunch of things (even though for the 2014.10 release these deprecations have been temporarily un-done again). Among others, she removed the ability of IO::Handle to represent an unopened file in this commit (somehow I’ve got the feeling I’ve talked about this one already…)
  • The already deprecated KeySet and KeyBag classes as well as the ucfirst sub and method have been kicked out of Rakudo as well as the .delete and .exists methods on Arrays, Hashes, Sets, …
  • moritz improved the worst-case speed of escaping strings in json by making the Str.trans method a whole bunch more efficient. On a string that’s just ten thousand newlines, the run time went from 13.5s to 4.9s after the first commit out of three.
  • FROGGS fixed parsing version literals that have a more-than-one-digit long major version number. This bug had been chilling out in our RT since Dec 2012…
  • b2gills fixed a bug with the enum constructor – that I stumbled upon when building the Cairo module – where giving an element in the enum a value by using a pair constructor with a quoted name wouldn’t set the counter for the next entries in the enum.
  • atroxaper added a –git-depth option to all the git clone commands in rakudo’s and nqp’s build systems. Especially the nqp repository is pretty darn huge due to all the old stage0 blobs we have in there…
  • brrt and me added some more operations to the JIT, as I’ve longed for higher “jitted percentage” in my shooter game thingie. Among them are savecapture, usecapture, getobjsc, and neg_n (by brrt) as well as sin, cos, tan, asin, acos, atan, atan2, time_n, nativecallinvoke (not as good as it could become, though!), pow_n and sqrt_n (by me). That day, I learned, that the jit C call emitter has to put floating point arguments into different registers than regular registers, causing my trigonometry ops to blow up quite spectacularly …
  • FROGGS worked with an OpenSuSE developer to improve the packagability of MoarVM.
  • jnthn vastly improved performance and correctness of the does and but operator with regards to roles and access to a metaobject’s methods. He also implemented a method “lookup” for metaobjects that will find a method for you that you can use for introspection (rather than invocation).
  • peschwa made the flipflop operator do a smartmatch against $_ instead of just calling .Bool.
  • After I found out that Bool.roll is about 10x slower than 2.rand.Int.Bool, lizmat went ahead and gave pretty much all picks and rolls a nice little overhaul that improves performance across the board. Nice!
  • lizmat also created a “new-from-absolute-path” constructor for internal use that ought to make path-juggling functions a nice bit faster, as well.
  • the “more” method on Supplies has been renamed to “emit” and the “winner” keyword has been renamed to “earliest”.
  • moritz and mouq did quite a bit of work on doc.perl6.org; Thanks, you rock!
  • tony-o started a project called “pandapack”. It currently does almost nothing, but I suppose we can already get excited about what it *may* do in the future :)
  • ugexe built an example .travis.yml that uses rakudobrew to compile a fresh rakudo and test your module.
  • Jnthn put his OO::Monitors and OO::Actors module up on the ecosystem – these are the modules he gave a talk about at the APW.
  • paul cochrane added PerlMongers::Hannover (inspired by Lynn Root’s pyladies lightning talk at EP2014) as his very first Perl 6 module. Welcome aboard! :)

Two more things worth pointing out that don’t quite fit into the bullet list format:

masak has been writing a whole bunch of posts about macros. These are meant to be kind of a “request for comments”, so feel free to point out things you find wrong/lacking/unsound/…

azawawi has been working steadily on improving Farabi6, a web-based IDE for Perl 6. The newest design is inspired by the way the IPython notebook handles code “blobs”. I kind of like it and I’m very eager to see what Farabi6 is going to look like in one or two month’s time and I’m hoping I can get a few patches in as well. I’m thinking it’d be a pretty lovely thing to have in Rakudo::Star! Here’s a screenshot for you:

A screenshot of the Farabi 6 in-browser IDE.

 

Even though this post is already quite a bit longer than the usual ones, I still feel like I’ve forgotten something :\

Ah well, it won’t be the end of the world!

See you around! :)

2014.40:

Hey, it’s timotimo again.

Last week, we decided to make building MoarVM with its JIT by default (thanks carlin!). It seems to be pretty stable, but now we’re going to get much more people trying it out.

Now it’s time to look through what else has been done recently:

  • lizmat has worked on IO::Handle and IO::Path stuff a lot. It used to be possible to have an IO::Handle that corresponds to a folder or be unopened. That means every single method on the IO::Handle first had to check if it refers to a directory or if it needs to open the file first. That was a major time sink, especially for cases like a while loop that gets individual characters from a file. It’s hard to put into words how much work this has been!
  • On top of that, lizmat also sprinkled lots and lots of optimizations all over the path/file related things.
  • FROGGS tracked down a major piece of start-up time wastage: on windows, rakudo took about 0.75 seconds to construct $?USAGE, even if it went unused. The new code is also faster than the old one used to be in case you actually do want to see the $?USAGE.
  • jnthn implemented the SUPERSEDE and DECLARE cases of EXPORTHOW.
    • SUPERSEDE lets you override the behavior of declarators like “class” or “grammar” in the scope your module gets imported into to use different metaclasses
    • DECLARE lets you create whole new declarators that act very much like the “class” or “grammar” declarators, like perhaps a “controller” declarator that adds a certain role to all things declared with that directive. (That example is what the initial tests in the spectest suite do)
  • Tony-O added a super simple ORM to the ecosystem called “DB::ORM::Quicky”.

Does that list seem not too impressive to you? Don’t worry, things are going to be exciting: This weekend the APW2014 will take place in Austria and it will be followed by two days of hackathon where many Perl 6 developers are going to be on site. Topics that will be covered include The Great List Refactor and there will surely be discussion of what topics and features to move past the “6.0.0″ milestone and what we’ll definitely spec/implement before calling that milestone complete.

I am going to be there, too. I will probably post a weekly that includes the things done at the hackathon and the things presented at the APW on tuesday rather than on monday, as monday is the last day of the hackathon.

In somewhat related news, the yapc eu videos from the Vitosha hall (which is where all the Perl 6 related talks were given) have still not been uploaded. Apparently the setup they used to record the talks is suboptimal and requires a lot of extra work or something like that? Sadly, I don’t have insider information for you :(

Oh and by the way: thanks for all the positive feedback in last week’s comments! I didn’t expect so many people whose names I’ve never seen before to speak up in such a kind manner :)

Looking forward to reporting cool stuff next time :)

2014.38 and 39: Star release and stuff

Hey there,

I finally broke through my little writer’s block phase (well, at least partially) to give you this two-week coverage post! Since it’s already a bit into the third consecutive week, I’m going to cover the 16th through the 28th of last month in this post.

In the first week, masak did a rakudo release and FROGGS did a Rakudo Star release with Parrot and MoarVM and the .msi files for Windows are available already as well.

A nice bit of news is that Rakudo-MoarVM now starts up about 15% faster than perl5 with Moose loaded. We need a different benchmark that puts us back in the second place so that we can improve some more ;)

Actually, on that topic, our performance is improving across the board, but let me break all the changes down in list form:

  • Thanks to lizmat, Supply.lines and Supply.words now exist and the lines method/sub for reading from files or splitting strings now take an “eager” option that gives a little preview of how fast things can get once we have the Great List Refactor™. Other than that, you can also pass a “count” parameter to get the number of lines instead of the lines themselves. Str.words got a similar treatment.
  • lizmat also made sure the output from a Proc::Async’s chars/bytes supplies will not give chunks out of order, which could sometimes happen before.
  • Another neat little thing lizmat added was the beginning of a “dd” sub that dumps variables’ contents together with their name to stderr.
  • There was an attempt to optimize the push method for iterables before, but it unfortunately led to a major performance regression in the variants we test in the benchmark suite. Jnthn now worked some more on that method and made it quite a bit faster than it was before.
  • FROGGS has already done the same optimization to rakudo’s MATCH method that nqp already got a while ago, but even though the initial version of that got optimized by 25% with a later commit, the performance hasn’t caught up to the performance it was before yet. We should reach a pretty good performance improvement after we work out the problems.
  • A big thing people have probably been waiting for is lax mode (not having to predeclare variables before using them). Thanks to FROGGS, you can explicitly enable/disable it with “use strict” or “no strict” (where “use strict” is still the default for scripts), but if you run your code on the commandline with the -e flag, you’ll have lax mode on by default.
  • The nativecallglobal op that lets you access globals exposed by a dynamically loaded C library is now also available on the JVM, thanks FROGGS!
  • Lots of spec test suite work has been put in by lizmat, dwarring, usev6, stmuk, TimToady, …

Here’s things in progress that have not really materialized yet:

  • Jnthn and masak were in China for a dayjob gig and weren’t able to get much coding in, but what they did work on is thinking and discussing a lot about how escape analysis should be implemented on MoarVM.
  • Another topic jnthn put some thought into is how to tackle the improvements to native values. Examples for this are being able to ++ a native int or having an “is rw” accessor for a native attribute in your classes or (which I’m really looking forward to) having native arrays in Perl 6 itself – which would also be a prerequisite to proper vectorization (think SSE) of hyper operations and such.

I would have loved to give you a quite thorough comparison of the performance of rakudo-parrot, rakudo-moar, rakudo-moar-jit and perl5 over more than a year, but across all the versions, all benchmarks seem to have missing data at least once, so the results ended up being “100 points every single release for every compiler, but no benchmarks were run”, which I thought wasn’t very insightful.

Anyway, I hope you’ll excuse the much belated and surprisingly short post; My energy levels seem to be below 0.009 recently :(

Have a nice rest-of-week, esteemed readers, and see you next week!

2014.37: Inline::Perl5, C Globals in NativeCall, …

Not terribly much seemed to happen this week, except that the pretty amazing Inline::Perl5 module progressed a whole lot. Here’s the changes in the list format you’ve likely grown accustomed to:

  • On top of the features you can already see in its readme, Inline::Perl5 can also handle exceptions thrown from p5 code into p6 code and exceptions thrown from p6 code into p5 code using the regular eval/try-catch you’re used to
  • FROGGS added support for “cglobal” to NativeCall, so you can access global values that are offered by libraries.
  • FROGGS also made sure MoarVM works with the latest libuv and bumped the revision.
  • FROGGS also added automated travis builds to MoarVM, NQP and Rakudo.
  • grondilu gave Synopsis 99 an overhaul making sure lots of broken links work now.
  • lizmat subjected Proc::Async to lots of testing and wrote some specs, so that people will have some documentation to read up on when they see the changelog entry for it that we neglected to mention in the last release …
  • lizmat also worked on Str.lines, making it a whole lot faster.
  • thanks to lizmat Str.lines and IO::Handle.lines now take an optional named parameter “eager” that makes them 5x and 3.5x faster respectively. But if you loop over these lines, you’ll have to wait for the whole split to finish before your loop’s body runs for the first time.
  • another thing lizmat added is the %ENV named parameter for Proc::Async to override the environment variables.
  • yet another thing lizmat added is a term π to refer to pi.
  • finally, lizmat made Bags and Mixes give Pairs instead of just the keys when evaluated in list context (like when you “for” over them).
  • cognomial implemented the :args parameter for Grammar.parse.
  • PerlJam built a more efficient version of “samecase” (which transfers upper/lower case information from one string to another)
  • jnthn taught the MoarVM profiler’s allocation counter to also count things allocated by extops (our mechanism for a language implementation (like Rakudo Perl 6) to add some extra opcodes to the VM).
  • moritz transferred mathw’s “Form” module to the perl6 organization on github. retupmoca sent in a pull request to make it pass its tests again. Maybe – now that it’s more visible – more people will feel like hacking on it.
  • krunen added a termios binding to the ecosystem. This allows for nice things like enabling raw input mode …
  • moritz fixed segfaults when trying to send data on a closed socket.
  • TimToady worked a bit on reification (which is asking a lazy list to generate values) to improve performance and fix a certain bug you may have hit if you asked a lazy list to generate a huge number of elements.
  • TimToady also implemented the plan method on lists, which allows you to append a second lazy list to a lazy list without forcing all items to be reified (which is what “,” would do)
  • On our mailing list, Alex Becker gave helpful tips for making Rakudo Star more useful and easier to get into for newcomers and got in a pull request for modules.perl6.org to be more user-friendly.

As for me … I didn’t do anything Perl 6 related last week.

But I may or may not be playing around with thoughts relating to a project I’ll give the abbrevation “RoR”. The rest is highly classified! :P

In related news, TimToady is “studying up” for the Great List Refactor, which is expected to make everything list-related much much faster soon™.

Anyway, I’m really looking forward to more performance tweaks and finally the GLR :)

Have a lovely week!

2014.36: Finalization, lazyness, optimizations.

Hi! It’s timotimo again and this week we’re pretty stoked to present some really neat stuff to you!

  • jnthn implemented finalizer support (giving your classes a DESTROY method or submethod that will be invoked when the object gets collected by the GC) on MoarVM and exposed it to rakudo. Now nine will be in a much better position to work on the Inline::Perl5 module!
  •  Speaking of Inline::Perl5, carlin contributed changes to the module that make it installable via Panda, so it doesn’t require any manual work any more.
  • brrt implemented a little function to force a GC run on MoarVM. It’s not meant for performance tuning, though. Its main purpose is to help write test cases that check the behavior of DESTROY methods and similar things.
  • MoarVM now properly reports errors when mallocing memory fails instead of just crashing, thanks to carlin.
  • carlin also fixed up some cases where signedness vs. unsignedness caused branches to always or never be taken in MoarVM’s guts.
  • brrt taught the JIT log to write out where inlines happened in the code.
  • I implemented bindattrs_* (bind a value to an instance’s attribute given a string that’s not known at compile time) and getattrs_* in the JIT.
  • MoarVM’s GC used to run a full collection through nursery and old generation every 25 runs. Now it counts how much data has been promoted to the old generation since the last time a full collection was done and bases the decision on that fact. I started the patch, jnthn gave it the finishing touches.
  • cognominal worked on the Synopsis 99 a whole lot. If you’re ever puzzled by lingo from the Perl 6 community (for example on this blog), this document lends you a helping hand.
  • The usual assortment of spec test improvements have also happened, of course.
  • In the ecosystem, leont contributed a TAP::Harness module and grondilu added Clifford, a geometric algebra module and btyler built a NativeCall binding to “discount”, a markdown compiler.

Another topic that got a lot of attention by the devs this week is performance of specific things that were currently implemented quite sub-optimally and gave users a very bad experience in common usage scenarios:

  • the .words method of Str used to use the .comb method to do its job. While this makes for very pretty and simple code, it’s far from optimal, especially since we know that whitespace and non-whitespace have to follow each other immediately, TimToady was able to produce an improved version that runs about 6x faster.
  • With well-placed native integers, lizmat was able to improve the speed of List.ACCEPTS (the method that a smartmatch turns into) by about 14% when the check needs to go through the entirety of a big list.
  • jnthn and lizmat went through a few global variables that have costly set-up routines that impact every single start-up. Now they will only be populated with data when they are actually needed.
  • Mouq gave us better code-generation for the -p and -n flags of rakudo.
  • Turning the methods .list, .flat, .eager and .hash of Any into multiple dispatch methods (implemented by lizmat) gives the optimizer better optimization opportunities.
  • lizmat also improved the implementation of “make” (which is used inside grammars and their action classes to attach objects to the match results) to not access the caller’s lexpad more than once. (I actually stumbled upon this opportunity)
  • The Whatever and HyperWhatever objects will no longer have new instances of them created whenever you write * or ** in your code. Instead, a pre-built singleton will be returned by each. This cuts a whole lot of allocations from all over. Especially when iterating over lazy lists you’ll end up calling .gimme(*) and .reify(*) often, where * is just used as a “signal”. These allocations are very small, though. TimToady and jnthn worked on this.
  • jnthn turned take, take-rw, succeed and proceed into multiple dispatch subs, which should impact situations where gather/take is in the hot part of the program and also positively impacts given/when statements.
  • A few micro-optimizations with a small — but definitely measurable — pay-off were done to Str.chomp by lizmat.
  • lizmat began improving IO.lines (makes reading /usr/share/dict/words line-by-line about 32% faster) and moritz added a 4% improvement on top.
  • moritz chopped off two container allocations for every call to ListIter’s reify. On top of that, a few containers in MapIter.reify had to go, as well.
  • By using codepoints instead of one-character-strings inside chomp, jnthn manages to remove some more string allocations in there.
  • jnthn also made throwing exceptions and returning more friendly to the optimizer (in particular, the inliner).
  • In a big bunch of commits, lizmat makes many dynamic variables lazily constructed, removing a significant chunk of the start-up time of every single perl6 program.

All it takes for the devs to start a little performance hack-a-thon is a good, reproducable benchmark with a comparison to another language.

Until The Big List Refactor has happened, the performance of lists, list iteration, lazy lists and so on are going to be suboptimal, as we almost always pay the price for a fully lazy list generation, even when the iteration can be identified to be eager.

Another thing to note is that the name “DESTROY” is not yet final. There’s some discussion about it still, because calling it DESTROY may give people familiar with perl5 a bad idea. As opposed to a reference counted implementation (like CPython or perl5), fully garbage-collected implementations (like PyPy and Rakudo) cannot guarantee that your DESTROY methods are called soon after the object becomes unreachable. The DESTROY method may not even be called at all if the interpreter shuts down before that particular object gets collected. If you want your destruction timely and guaranteed, you’ll have to “do it yourself”.

Either way, it’s been a great week for both performance and feature advances. I’ll be back in a week, or maybe I’ll run benchmarks tonight and just publish an out-of-schedule post here with my results :)

See you around!

2014.35: A Star Release, perl5 interop, …

Hey, timotimo here with another little Perl 6 update!

This time, I can announce an actual Rakudo Star release. FROGGS did the honors this time. The release announcement contains a few ChangeLog entries, but since the previous Star release was in 2014.04 there’s a whole lot of changes you’ll get to enjoy. If compiling the Rakudo Star stuff by yourself seems like too much of a hassle, hoelzro’s got you covered with his Docker image of Rakudo Star.

A few of the YAPC::EU videos have been uploaded to youtube. It seems like these are only the videos from one of the three rooms. None of the Perl 6 related talks happened in that room, though.

After the YAPC::EU, Ovid published a nice blogpost entitled “Try rakudobrew and play with concurrency”.

Other than that, a few things have happened since last week:

  • nine continued work on the “embedding perl5 inside Perl 6″ project. On tuesday he announced he could now use DBI inside perl5 inside rakudo. An important feature for making this work absolutely cleanly is still missing: DESTROY submethods on the Perl 6 side don’t get called by the GC yet, so you’d have to manually manage the Perl5 objects in your program.
  • Jnthn implemented the mechanism to give objects a persistent WHICH when they are used as object keys. This feature is only available on MoarVM, though. On Parrot it is not necessary, as parrot’s GC doesn’t move objects. It will have to be implemented on the JVM backend at some point, though.
  • Jnthn utilized the profiler he built recently to optimize the MATCH method in NQP, which gets called whenever a regex (for example, any token/regex/rule inside a Grammar) matches something and a Match object has to be created. They used to allocate a whole lot of Boxes that were immediately thrown away.
  • jnthn and TimToady worked on making errors like using say without an argument happen at compile-time; it works for much more than just “say”, too!
  • In order to further make matching faster, brrt and I implemented some more ops in the JIT that were blocking frames from being jitted (these include bindattrs_*, indexat and indexnat)
  • lizmat invested a whole lot of time into many different WHICH methods (which are used to create keys for hashes and compare things for equivalence)
  • lizmat also improved the performance of path-related operations and rakudo startup.
  • TimToady implemented many cases of anonymous variables
  • TimToady also started on an implementation of the plan method for lazy lists, which allows you to add a list to the end of a lazy list without forcing it to be eagerly evaluated.
  • hoelzro put more work into the Pod documentation implementation of rakudo.
  • nwc10 improved MoarVM’s compatibility with PowerPC (mostly endian-ness related changes that’d also apply to other architectures)
  • carlin fixed a whole bunch of warnings clang would point out in MoarVM and added arc4random support on platforms that support it (like BSDs)
  • carbin and vendethiel added Crypt::Reop and Sprockets respectively to the ecosystem.
  • masak advocated (and finally implemented) a change to allow *..* as ranges (which mean the same thing as -Inf..Inf)
  • there’s been work on the spec test suite by hoelzro, lizmat, TimToady, jnthn, masak, FROGGS, and Mouq

All in all it’s been a good week. A friend of mine lent me a book about x86 assembly and I’d love to learn what SSE has to offer and maybe implement a simple hacky prototype for arrays of int or num that can be used with SSE instructions or something …

Anyway. Hope you’ll have a nice week :)

2014.34: chugging along, GSoC is over, YAPC::EU, …

This week, there was a Rakudo release (not Rakudo *, which is the distribution that contains the compiler and a bunch of modules) and for someone who usually runs the latest master branch code, seeing the change log is always pretty neat.

Over the weekend the YAPC::EU 2014 happened in Sofia, Bulgaria. There were a few Perl 6 talks as well as lightning talks. I heard the video recordings will be up in “a matter of weeks” … Jnthn did a talk on concurrency and a talk on Rakudo performance advances, in which he revealed he’s been working on a profiler in secret. That was a pretty “wow” moment for pretty much all of the Perl 6 community! There was also a talk by masak about goto and why it’s awesome (no, really) and a cute little lightning talk. On top of that, there were talks by lizmat and lichtkind and a lightning talk by nine, who showed embedding a perl5 interpreter inside Rakudo and embedding Python code inside that. Nifty!

Sadly, I missed many of the talks, so I’ll be patiently (or impatiently) waiting for the recordings to be published.

Anyway, here’s your summarized list of changes during that week:

  • TimToady continued work on making our lazy lists less costly in eager context, like making the xx operator many times faster, so it’s now useful for filling very big arrays or lists with some initial values.
  • Some more performance tuning happened to the X operator (especially for ranges, which you’d idiomatically use to replace a nested for loop to go over 2D coordinates, for example), and the combinations and permutations methods. More is very likely to come soon!
  • Together with brrt, I implemented spesh’d operations as well as jit output for the boolification of iterators; If we already know we’ve created an iterator from an Array or from a Hash, we don’t have to check if the iterator is an array or hash iterator every single time we boolify. And since loops over arrays and hashes in NQP usually end up as a while loop that checks the iterator it creates for its truth value, this ought to show up in a bunch of hot loops.
  • A huge change (which I mentioned above already) is the profiler that got added to MoarVM. You can just run your perl6 or nqp code with –profile or –profile-compilation and get a nice little html output that you can comfortably browse through. Neat!
  • hoelzro and sergot have been working some more on making HTTP::UserAgent style things better/more pleasant/more powerful/more extensible/…

This list is already it, but especially the profiler item is a huge thing. Many of our devs were attending the YAPC::EU event, so they have been watching talks and socializing instead of coding all day and night.

Another thing is, that it’s always quite a bit of work to prepare the release, make sure the test suite remains clean, have your release candidate tarballs tested by enough people, … – I’m glad to say that Coke did a wonderful job this month. I’m hopeful we’ll see a new Rakudo Star release this month, as problems between Parrot and Rakudo that prevented last month’s release have been worked on in the mean time.

Of course, with rakudobrew being readily available as a simple way to install panda (our package manager) for you, the need for a Rakudo Star release is not really that big if you’re willing to put a few minutes rather than a few seconds of work into getting a running Perl 6 compiler.

Another thing: Google Summer of Code is now over and the evaluations have been sent in. Both brrt and sergot have passed and we are extremely pleased with how both projects have turned out so far. In addition, we’ll see more contributions from brrt and sergot in the future :)

Finally, if you’re as impatient as I am, you can have a look at different slide sets that were presented at YAPC::EU.

Anyway, that’s it for today. Hope you’ll have a nice week :)