2015.29/30 FastForwarding To The Present

After having gone back a year two weeks ago, this Perl 6 Weekly should bring us back to the present and up to speed.

Blog Posts

The past 2 weeks have seen quite some interesting Perl 6 related blog posts:

2015.07.2 Prague

The past week also saw the monthly compiler release of Rakudo, named “Prague“, a joint effort by Carl Mäsak and Moritz Lenz. It came in at about 350 commits. There is also a new Rakudo Star release (by Moritz Lenz).

Multi-Lingual Pun Of The Week

* cdc veut un racadeau pour Noël.

More Asynchronous Sugar

Jonathan Worthington wrote up his thoughts on more syntactic sugar and improved semantics for asynchronous execution. New goodies:

  • await – more powerful with a dynamic Awaiter
  • serial supply – promises you will never do a concurrent emit/done/quit
  • supply syntax construct – create on-demand supplies more easily
  • whenever – an asynchronous loop construct
  • Channel – should probably be renamed for Concurrent::Queue or some such
  • earliest/more/wait/done to be deprecated

Check it out and comment if you want!

Perl Presence at OSCON 2015

The past week saw OSCON again, for the last time in Portland, OR. Perl’s presence there was unfortunately limited. Community booths were smaller again than the year before, and there were fewer of them and they were hidden at the back of the Expo Hall. In a keynote the lane with community booths was referred to as Diagon Alley. It would therefore seem that OSCON is becoming more and more a muggles-only event. OSCON Europe 2015 is said to have no community booths at all.

Anyways, a Perl 6 introductory tutorial by Damian Conway: Perl 6: – Transparadigm programming 101 saved the day in Portland. It also came with script examples. The other presentation by Damian Conway was: Don’t change languages. Change your language. Here he introduced the very nice Perl 5 module Dios Declarative Inside-Out Syntax, which more or less gives you Perl 6 class syntax in Perl 5. Damian also introduced modules that were more oriented towards Perl 5, such as experimentals and Keyword::Declare. If you’re still using Perl 5, they’re definitely worth looking at.

Culling from the Backlog

Recent additions to the Ecosystem

Two weeks worth of additions gives us:

  • Ini::Storage Read/write ini files and manipulate them in memory (by jaffa4)
  • Audio::Encode::LameMP3 Encode PCM Audio data to MP3 in Perl 6 using a binding to liblame (by Jonathan Stowe)
  • AVRO support (by Adriaan Larmuseau)
  • Log::D logging (by jaffa4)
  • Stream::String Stream into a string or from a string (by jaffa4)
  • Atom::Electron Perl 6 desktop applications using JavaScript, HTML and CSS on top of the Electron platform (by Ahmad M. Zawawi)
  • Data::Selector data selection dsl parser and applicator (by Justin DeVuyst)

Winding down

That’s it for these two weeks again. Hope to be having the next one on time again. Have a lot of -OFun!

2014.28: Title goes here. Fill out under all circumstances!

Hello everyone!

Timo is handling the metaphorical pencil again – that’s why the weekly is so late once again …

Well, let’s just see what last week had in store for us. Here’s a few articles to set the mood:

And here’s a bunch of changes – I’m pretending it’s still monday and I’m leaving out the changes that already happened, so this will cover everything between the 6th and the 13th:

  • skids improved the .assuming method on callables (that you can use to get a partially applied version of an existing sub) to generate proper signatures for the resulting callable.
  • lizmat improved the “splice” method on lists to work a bit better; emptying a list (by using splice without arguments) is now 6x faster, for example, and infinite lists are no longer allowed to be spliced in – they probably caused an infinite loop before.
  • thanks to lizmat and moritz, Set and its many friends can now be subclassed by user code again. There used to be a little hack in place that caused errors when a derived class tried to BUILD.
  • In our documentation, smls documented the “fail” sub – and I almost put in some more things here that have happened between monday and today. oops!
  • in MoarVM, we didn’t care to free strings that were used to build error messages, which would then leak. hoelzro has committed quite a big patch that uses our relatively new “throw exception and free strings” function to prevent leaks.
  • brrt made MoarVM build properly with ASAN again on linux; something I believe broke when FROGGS added the choice between using dyncall and libffi.

And here’s a little list of ecosystem additions:

  • BioInfo by MattOates, a BioInformatics package built to benefit from fresh Perl 6 features.
  • Algorithm::Elo by hoelzro, an implementation of the Elo ranking algorithm
  • Dependency::Sort by jaffa4 for turning directed graphs (of dependencies for example) into a linear form

And that’s already it for this week! But new changes have already started accumulating, so see you again next week!

2015.27 It was HOT

Perl 6 Development Fund – Grant Status Update

Jonathan Worthington has written a Grant Status Update in which he lists his major achievements of the grant work so far. Sadly, as it stands now, the end of that grant work is in sight as funds are running out. You can support the Perl 6 Core Development fund yourself (select Perl 6 Development), or convince someone in your organization to have your organization become a major sponsor!

Mogrification in Progress

Jeffrey Goff has taken Perl::Critic and mogrified that into a Perl 5 to Perl 6 Translator. It is a very interesting work in progress, worthy of checking out!

FatRats divide Better

Solomon Foster blogged about a Fun and Easy Fibonacci Trick. FatRats for The Win!

Intermediate Progress on an Intermediate Representation

Bart Wiegmans takes us down into the rabbit hole that is implementing something very much like DynASM. Since that wasn’t part of the original plan at all, Bart thought it justified an explanation.

Using Xapian with Perl 6

Rob Hoelz made an excellent screencast about how he integrated the (C++) Xapian library using NativeCall. One can only hope for more screencasts to come!

AmsterdamX.pm YouTube Channel

Upasana Shukla has announced a YouTube Channel for presentations made at AmsterdamX meetings. Two presentations are of interest for this reader group:

The Perl 6 Story

The Perl 6 Story is another of Andrew Shitov‘s projects, with a lot of nice interviews. Some of which have been mentioned here before, and some have not. Most notably not mentioned here before, are interviews with Carl Mäsak, Stevan Little and Flávio Glock.

From the Development Front

  • Jonathan Worthington has created the underpinnings to multi-dimensional arrays in MoarVM (which e.g. lets a 3x3x3 int Array exist as a single packed blob in memory). The Perl 6 interface to this functionality should become visible in the coming days.
  • Larry Wall tells us he’s been privately prototyping the new non-flattening list and array semantics to see where the new gotchas will be.
  • Leon Timmermans unbitrotted Tap::Parser and added a synchronous TAP parser as well!
  • Elizabeth Mattijsen added a no worries pragma to surpress those pesky “there could be something wrong here” warnings, mostly intended for programmers migrating from Perl 5. She also added two of these warnings (internally called “worries”) for use of the backslash in a Perl 5 way.

From the 4 week backlog

Recent additions to the Ecosystem

In the future, I would like to show the new additions of the past week to the Perl 6 Ecosystem. Since this is the first time, I’ll list them here since the 1st of June:

Winding Down

That’s it for this record-breaking hot week (at least in this part of the world). Hope everybody will keep their cool in the next one!

2015.26 Half-way through the year

Some exciting blog posts came our way this week:

Blog Posts

Rakudo Star 2015.06

There is a new version of Rakudo Star available. Kudos to Tobias Leich for taking care of this!

London Perl Workshop

The London Perl Workshop 2015 has been announced: it will be held in central London on the 12th of December. Its theme will be “A Perl Christmas(?)“. If you have anything to show of Perl 6, and you want to show it in London, be sure to propose a presentation!

Building a Perl 6 Project with Travis CI

Travis-CI now supports Perl 6 out of the box, check out the documentation. Kudos to Paul Cochrane, Rob Hoelz, Nick Logan and Tony ODell for making this happen!

Remarkable Changes of the Week

Apart from many smaller fixes and improvements, these improvements are of note:

  • Adding “is required” attribute trait (by Will Coleda). So now you don’t have to do the ugly
       has $.foo = die "'foo' must be specified";
    anymore if you want to indicate that the specification of a given attribute is mandatory.
  • Deprecation of pipe($cmd) in favour of shell($cmd,:in,:out,:err) by Tobias Leich, as mentioned in the previous Perl 6 weekly. This allows you to read stdout/stderr from run() and shell() by passing :out and :err, and you can write to its stdin by passing :in. You can also suppress e.g. STDERR from a subprocess by passing :!err.

Catching up on backlog

It looks like I’m now behind 4 weeks of backlog, instead of 3. Hope to find the inspiration to go through them in the next Perl 6 Weekly! Have a good one until then!

2015.23..25 Catching Up Is Hard To Do

The past weeks we’ve had no Perl 6 weekly. Trying to catch up with so many things happening on-line and IRL is quite a lot of work and prone to omissions. Please bear with us while we get things on track again.

Blog Posts

For those of you who missed them, there were some excellent blog posts in the past 3 weeks (in chronological order):

Conferences, Conferences, Conferences

The conference season has really started again. Some Perl 6 related videos from the German Perl Workshop in Dresden:

Some Perl 6 related videos from OSDC.no in Oslo, Norway:

And some Perl 6 related videos from the YAPC::NA in Salt Lake City, Utah:

And some related blog posts:

The GLR Is Spreading Its Wings

Patrick Michaud has started a new version of the List And Iteration Synopsis. If you want to keep abreast with the developments there, please check it out. If you have any comments, let them be heard now rather than later!

Rakudo 2015.06 Was Released

Rob Hoelz made the Salt Lake release happen, which came in at 200+ commits. Since then we have seen about 40 commits for the next release, mostly consisting of bug fixes, performance enhancements and the odd new deprecation.

Busy FROGGS

Tobias Leich has been busy. He made:

  • Rakudo on MoarVM buildable on Windows with gcc/gmake
  • Use run()/shell() as a replacement for pipe()
  • Added libffi support, which should let build MoarVM on all platforms

This should make MoarVM build on all Debian platforms except Hurd.

Unicode-8 Support Is Now In

As of 23 June, Rakudo on MoarVM supports Unicode 8.0, only about a week after its release! Appropriately for the Perl 6 Party, it has a code point for Bottle With Popping Cork!

Covariance And Contravariance

A Fresh Look At An Old Issue

Giuseppe Castagna has written an interesting article about (static) typing for Perl 6:

The target reader of (the first part of) the article is every Perl 6 programmer, which is why all the above is explained by examples of Perl 6 code (no formula, theory, theorem, or property whatsoever) and I tried to do my best to make the article completely self-contained…. The second part of the paper targets language implementers and designers.

I haven’t really gotten around to reading it thoroughly, but I find it very interesting that all examples in this article are using Perl 6! Giuseppe is welcoming comments on the per6-language mailing list.

Winding Down

Perhaps this Perl 6 Weekly was a bit too sparse on events on the #perl6 channel. Trying to catch up with 3 weeks of backlog has become such a daunting task, that I have only scanned the first week of June for interesting bits (so far). If there are any other interesting bits in the backlog in weeks 24/25, I will mention them in the weekly of week 26.

If you happen to be in Karlsruhe this weekend, find me at the GPN15!

Hello, dear esteemed readers!

I’ll be attending the Gulaschprogrammiernacht 15 in Karlsruhe this weekend (starting this Thursday evening already), so if you’re interested to talk about Perl 6 or maybe even write or port a library or something, I’ll gladly give you a portion of my time :)

If you’re in Karlsruhe and are not attending, that’s no problem, as there’s no entrance fee, so you can just teleport on by and see what’s up. You can find out more about the event at http://gulas.ch/.

If you want to meet up, you should be able to reach me on the freenode IRC network on #perl6, which I’ll likely check regularly.

See you there, or maybe not!
– timotimo

2015.22: While async & conc are still a bit crashy, enjoy these unrelated improvements …

Last week’s blog post by masak about the “Send More Money” mathematical puzzle kind of sent me off on a little coding spree. A quick profiling run of the “loops” version of the solution quickly showed that a sizable chunk of the run time was spent inside the implementation of the “next” keyword (well, sub really.).

After just a bit of fiddling around with the implementation of “next” – which had been pessimized some time ago for some reason – together with lizmat, the benchmark went from 22s to 15s, about a 30% improvement. This is not only purely less time spent running, but also less work done GCing.

Not only “next”, but also “redo” and “last” have been improved in the same way.

lizmat also went ahead and added a :as parameter to the categorize method like the classify method already had. This lets you change what the value that came in will end up as in the resulting hash. On top of that, the classify and categorize subs also work with a :into named parameter, which allows you to supply the exact type or instance to put the values into.

hoelzro put on his grammar hacking robe and NQP hat and improved error messages for cases when constructs get opened, but not closed. See bugs #125247 and #125153 on the RT.

Another thing hoelzro made work was the .push method for the Buf class (buffers that contain integers of a fixed size, closely packed together).

And then there’s also an interesting fix where an array of just a single pair in adverb notation would have the parser attempt to parse it as a reduce operation. Here, look:

[+] 1, 2, 3   # reduce operation
[:a]          # an array with a single Pair in it

This used to blow up, but works now.

Something I did was dump more information about logging and guards in the spesh dump log. I also found a bunch of places where creating P6int instances didn’t go through our integer cache that hopes to reduce the number of different incarnations of numbers 0 through 15 (now -1 through 14).

Finally, here’s a nice short blog post from domm about his experience porting a very simple script from perl 5 to Perl 6.

Looking forward to what the next week will bring :) In any case, I hope yours will be fine :)

2015.20&21: Blogs, Grant, Release and Progress

The past weeks we’ve seen quite a few nice Perl 6 related blog posts. Just in case you haven’t seen them, I would like to repeat them here for you:

The Annotated Berlin Consensus

At the 2015 QAH in Berlin, a consensus was reached on a number of Perl toolchain issues, with a particular focus on governance and recommended standards of care for CPAN authors. The official version in Markdown can be found on Github. David Golden also blogged an annotated version of it. Although one could argue this is not strictly Perl 6 related, all of the issues mentioned in the consensus, will come to the Perl 6 ecosystem sooner or later.

Advancing the MoarVM JIT

This grant proposal by Bart Wiegmans (brrt) has been accepted. This is great news, can’t wait for bare Rakudo startup time to drop below 0.05 seconds!

Bare startup time below 0.1 second

In the past week, the bare startup of Rakudo on MoarVM finally dropped below 100 milliseconds on most developers’ machines. This is now only a factor of 20 slower than Perl 5. However, I’m told that starting Perl 5 with Moose (which from a feature point of view, gets it closer to Perl 6), is now slower than starting Rakudo!

Dresden

We also had the 2015.05 compiler release of Rakudo, named Dresden (after the recent German Perl Workshop there). The list of changes is quite extensive, coming in at 370 commits since the last release.

Changes since the Dresden release

  • use trace: lexical pragma to trace program execution on STDERR
  • Empty IO::Path / “”.IO no longer allowed, it used to be equivalent to “.”

GLR underway, watch your modules!

One can say that the GLR (Great List Refactor) is now really underway. Several modules in the ecosystem are now known to fail because of GLR induced changes. If you have a module in the Perl 6 ecosystem, it would be nice if you could check it for GLR induced bitrot using the latest Rakudo!

Shutting down

This Perl 6 Weekly came in a little late. Hopefully, future Perl 6 Weeklies will come in a more timely manner. Until then, keep up the good work!

2015.19: Events & Conferences

During the last week, there were multiple conferences where perl6ers could be found:

  • Larry Wall gave a talk at Collision in downtown Las Vegas and a Q&A session afterwards using twitter and an app called pundit. I have no idea if there’ll be recording of the talk, but supposedly, you’d be able to get at the Q&A stuff somehow somewhere …
  • At the German Perl Workshop froggs talked about native library bindings, using libxml2 as an example.
  • Also at the GPW, lichtkind gave a talk about functional programming in perl6, but the slides are in german.
  • Another talk from the GPW is rurban’s overview over VMs related to perl. The talk and the slides are in english.
  • At the OSDC.no there were many perl 6 related events, which can all be found on the “list of talks tagged Perl 6“. Sadly, no links to slides can be found there, and I don’t yet know about recordings.

But there was also some work being done – there were post- and/or pre-conference hackathons, too.

  • froggs threw pretty much all files that were in our roast repository into rakudo’s spectest.data, which means that a whole bunch of test files weren’t actually being run until now. Those are all fudged properly now, though, so that we can further decide to throw out tests that haven’t stood the test of time, for example. [correction: actually a bunch of those are still not being run, but at least got triaged]
  • TimToady added line number annotations to warnings more often, and also hunted down a bunch of “leaking” Failures, which means a Failure gets created at some point, but never sunk or inspected. (remember, Failures are basically unthrown exceptions)
  • lizmat has improved @*INC and the “use cur” pragma (which is going to replace “use lib” when it works as wanted – of course it’ll also get the “use lib” name at that point) a whole lot.
  • TimToady is moving forward to make Nil no longer disappear when doing list flattening or assignment. Right now, there’s an Empty value that’ll do the “disappears in flattening” and Nil no longer disappears in lists. If you yourself want this behavior, please use () instead, as Empty is most probably going to disappear once the “switch-a-roo” is done :)
  • thanks to raydiak “undeclared variable” errors now show a more correct error position and don’t give you the list of what kinds of rules/tokens the parser was expecting at the point of failure
  • lizmat restored the performance of IO::Handle.words and .lines when providing the :eager parameter, that you can use to not pay the “lazyness cost”. Of course, this is what we want the GLR to “automatically” provide in most cases: not paying the lazyness cost if you’re actually eagerly using something that can be lazy if it wants to (or if you want to).
  • jonathanstowe improved the “earliest” block which now handles having both “more $foo” and “more *” blocks in it.
  • brrt gave me the go-ahead to merge the “jit_devirtualize_reprops” branch, which turns indirect calls to a bunch of operators on objects into direct calls (if the type is known at jit-time). This is also something we’d have wanted for brrt’s sponsored work on improving the MoarVM jit further.
  • froggs is quite far on the way towards getting “ignoremark” support into rakudo, which lets a regex (or parts thereof) match regardless of any combining characters applied to the target string. [correction: in fact, froggs is so far along the way that ignoremark is already in rakudo!]
  • jonathanstowe added some more concurrency related docs to our documentation repository (and website).
  • moritz described how to get default values at object creation time into your classes in the classtut

That’s all I have for now. There wasn’t quite as much progress with so many perl6ers traveling and attending or giving talks and workshops.

And at 666 words (not including this paragraph), I feel this is a good place to stop writing. Have a nice week, everybody!

2015.18: Startup time rectified, hashes unordified, startup speedupified, puns deduplified

This week pleases me greatly, because there’s performance improvements to report!

Startup time & performance

lizmat worked hard to make initialization of the module installation database in rakudo lazy, so that any script that doesn’t “use” any module won’t take the performance hit. This (together with a bunch of other improvements) gets startup time (as measured by compiling and running an empty program) back to 0.17s on my machine (previously it was 0.31s) and the memory usage went from about 85 megs to 69 megs (these numbers come from the time command outputting the maximum resident set size). On lizmat’s machine, that reduced the time it took to do a full spectest run from 1620 to 1435 CPU seconds.

On the other side of the module installation database coin, froggs and jnthn just had a little breakthrough that will finally allow the json database to be replaced with a MoarVM serialization blob. Needless to say, this will be a whole bunch faster to load and store when we actually do need to load the database.

On top of these big improvements there’s of course been small improvements again. After profiling found out that a lot of time is spent decoding utf8 during startup, jnthn switched a bunch of things that are known to always be valid latin1 to decode that instead. This reduces the instructions run during startup by about 10%.

Another nice change is that .moarvm files used to contain not only a string heap section but also a chunk of code that used to create a list of all these strings. Now the string heap is used for both purposes, shaving a good 60 kilobytes off the CORE.setting.moarvm (still about 9 megabytes big) and reducing the startup time by about 2%. And then there’s a few sundry changes that all shave off a few percent here and there.

Adding to this heap of performance improvements all around, hashes on MoarVM no longer preserve insertion order. This feature – which users were discouraged from relying on anyway – was implemented in uthash by a doubly-linked list. jnthn removed that doubly linked list and made hashes save a noticable bit of memory: 16 bytes per hash entry and another 8 bytes per hash.

In the future, uthash will be more heavily modified so that strings representing the same text in different storage formats (8 byte per character vs full NFG) will result in the same hash. Then we’ll finally be able to store simpler strings more compactly. I’ve been looking forward to this for a long time, and now it seems to be within reach :)

Something I forgot to mention last week is that a whole bunch of methods on Str are now much faster because they have gotten their own implementation rather than relying on Cool to implement them (and do a needless coercion first in every case). This change was inspired by making .chars much faster in the very same way.

Bug Fixes

There used to be a bug where you’d get maddening errors like this:

Calling rl_get_screen_size(TypedPointer[Int], TypedPointer[Int]) will never work with declared signature (TypedPointer[Int], TypedPointer[Int])

This came from NativeCall using the “make_pun” meta-object method that erroneously created a fresh, distinct class each time TypedPointer[Something] was used somewhere. That method call has now been replaced with just “pun”, which goes through a cache first.

ptc found and fixed a few small memory leaks in MoarVM.

Today jnthn spent time on known multithreading-related instabilities and oddnesses. A big part of that was fixing a race in the Fixed Size Allocator which is used for frames, on-stack-replacement, stuff like that. There was a CAS-based lock that suffered from an ABA problem, which basically means one thread interpreted “two changes” (there and back) as “no change” (“go ahead”) and the list of free spots in the allocation pool could get corrupted, leading to memory corruption in objects when two threads use the same spot for different types of objects. The multi dispatch cache also just got a lock added to it, too. There is still more waiting to be found and fixed, though.

I personally got a tiny bit of work in, as well: If you ever encountered the error “Can’t adverb that!”, you’ll now find that it’ll also give a little hint as to what exactly the parser thought you were trying to adverb (because the precedence level of adverbs takes a little getting used to, IMO).

There used to be a problem related to statement modifiers and the given and for keywords where the value of $_ would get messed up, and block-less “gather” sometimes didn’t work where just adding curly braces would fix the problem. Thanks to jnthn, that’s fixed now as well.

Implemented features

Thanks to lizmat variables can now take “will” blocks, which correspond to phasers. Those already implemented are enter, leave, keep, undo, first, next, last, pre and end. Also, unimplemented “will” types now throw an error instead of failing silently. In addition to that, “lazy attributes” are now supported in rakudo. These let you write code like this:

class Example {
    has $.attr will lazy {
        # do some expensive set-up here
        "hello" # value returned from this block will be used
    }
}

When the attr is first accessed, it will run the code to set up the value and after that it will behave like any regular attribute.

After making List.roll faster last week, lizmat also made List.roll(*) (and List.roll(Inf)) return a list that knows it’s infinite.

If you set the environment variable RAKUDO_VERBOSE_STACKFRAME to a number, that number of lines of code around a line indicated in a stack trace will be shown. Another change to stack traces is that instead of a few select routines from the core setting, all routines in the core setting will now be hidden from backtraces by default – you can get full backtraces with the –ll-exception flag as before.

Even though this isn’t strictly a feature, there’s also been more changes regarding the GLR – mostly how flattening and nonflattening happens in relation to “for” and “map” in both their sub and method form. Lists and Parcel have been moving closer and closer and soon the Parcel class will be thrown out.

Ecosystem goodness and documentation

Even though recent changes to the module installation system and making hashes unordered brought a few bugs and problems out of hiding, there’s more nice stuff to be found in the ecosystem now!

  • Module::Minter gets your own module started with the basic files every such project needs (by dnmfarrell)
  • Template::Protone lets you write templates that compile down to fast perl6 subs and let you easily embed perl6 code (by tony-o)
  • ANTLR4::Grammar lets you parse an ANTLR4 grammar and use it to parse stuff. (Though it’s not been put into the ecosystem yet, I decided to highlight it here none the less.) (by DrForr)

ptc has pushed a whole bunch of commits to the perl6 examples site and ptc, stmuk, jonathanstowe and moritz have been adding and modifying more and more stuff on the perl6 documentation site.

labster wrote a blog post “Three Tales of Second System Syndrome” that he put up on his site, hoping to get a bit of feedback before spreading it further. However, it reached Hacker News only shortly after he went to bed and there’s now 121 comments in the corresponding thread.

That’s it for today!

Very strange; even though it feels like I’ve written the longest post in a few weeks, I also have a sneaking suspicion that I’ve forgotten to mention multiple things!

In any case, it’s time for me to finally finish up and maybe get back to annotating MoarVM’s code with a few hints for Valgrind and its tools. Have a great week!